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identify path 



determine graphics guide thst provides substantially 
unobstructed view of instrument during positioning 



T 



construct graphics guide to frame path 
(with a vi/indow for steps 7303-b, an opening for step 730c 
or a pattern of lines for step 730d) 



730a 



730b 



construct polygon that 
has 2 opposing ends 
a nd window from one 
end to the other 



construct partial cylinder 
that has first and second 
ends and window from 
first to second end 



710 



720 



730 



730c 



construct circle with 
opening (to resemble 
letter C) larger than the 
width of the member 



730d 



construct pattern of lines spaced 
to allow member to pass through 
at least 2 consecutive lines (of 
optionally different spacing or 
length with respect to other lines) 



J 



track observer and/or instrument, 
render graphics guide (e.g., solid, semi-transparent, textured, 
outlined, or combination thereof) in alignment with the path 
including orienting guide with respect to observer, and 
optionally render representation of the instrument 



750a 



orient polygon 
to have 

centerltne of 
window aligned 

with the path 



750b 4" 



750c1 



orient partial 
cylinder to 

have length- 
wise axis 

aligned with 
the path 



735. 740, 750 



orient circle to 
substantially enclose 
target location, with 
opening to the right for 
right-handed observer 
and to the left for left- 
handed observer 



dynamically 
adjust/re-adjust 
orientation of window 
relative to vantage 
point of observer 

760ab 



1 760c2 


750<j , 


r 


orient circle to 
substantially 
enclose target 
location, with 
opening enclosing 
an axis of the path 




orient pattern to 
surround target 
location and with 
the at least two 
consecutive lines 
framing the path 



dynamically 
adjust/re-adjust 
orientation of opening 
relative to vantage 
point of observer 

I 760c 



dynamically 
adjust/ re-adjust orientation 
of pattern of lines relative 
to vantage point of 
observer 

I 760d 
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start ^ 



determine a graphics guide for 
positioning an instrument 



1102 



render the guide such that an 
appearance of a portion(s) of the guide 
is modulated with respect to space and/ 
or time 



1104 



1104a 



vary a transparency of the 
portion(s) with respect to other 

portions to provide a 
substantially unobstructed view 
through the portion(s) to the 
instrument (or part thereof) 



1104b 



vary a transparency of the 
portion(s) during pre-defined 
time intervals to provide a 
substantially unobstructed view 
through the portion{s) to the 
instrument (or part thereof) 
during the pre-defined time 
intervals 



Q end J 



1104c 



sequentially vary a 
transparency of portions (e.g., 

consecutive or non- 
consecutive) during pre-defined 

time intervai(s) to provide a 
substantially unobstructed view 
through each of the portions to 
the instrument (or part thereof) 
as the transparency is varied 



FIG. 11A 



^ start ^ 



determine a graphics guide for 
positioning an instrument 



1112 



construct the graphics guide as 
a line or cylinder 



1112a 



render the guide such that an 
appearance of a portion(s) of the guide 
is modulated with respect to space and/ 
or time 



1114a 



modulate a transparency of the 
portion(s) with respect to time 

so that the line/cylFnder 
repeatedly fades in and out of 
view to provide a substantially 
unobstructed view of the 
instrument when the line/ 
cylinder is out of view 



1114 



1114b 



modulate a transparency of the 
portion(s) so that a part{s) of 
the instrument is substantially 
unobstructed when viewed 
through the portion(s) 



1114c 



modulate a transparency of the 
portion (s) with respect to time 
and space so that at least a 
porion of the instrument is 
substantially unobstructed 
when viewed through the 
portion(s) during pre-defined 
time intervals 



T T T 



FIG. 11B 



1200 




FIG. 12 



1300 




1320 / 

1310 



FIG. 13 



1300 




1470 1430 



FIG. 14 



Q start ^ 



establish a viewpoint, from which a line of sight 

to a point on a target defines a path for an 
instrument to follow during a positioning of the 
instrunnent to the point 



1802 



optionally render the target as a graphics object 



1804 



mark the path with a plurality of graphical 
markers 



1806 



align the instrument along the line of sight to the 
point on the target 



1808 



position the instrument tip on the path 
at an entry point on a surface of a 
physical object corresponding to the 
target 



1808a 



rotate the instrument around the tip on 
the entry point until the instrument is 
aligned with the line of sight toward the 
point on the target 



1808b 



move the instrument along the path towards the 
point on the target 
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^ start ^ 



render a graphics path guide(s) for 
indicating a path for an instrument to 
follow to a target 



1842 



Identify an axis nnarker on the 
instrument that identifies an axis of the 
instrument disposed from the front to 
the back of the instrument, the back 
facing the user 



1844 



4 1846 

construct and render 
the axis marker(s) as a 
graphical marker, and 
track the instrument to 

register the graphical 
marker{s} with respect 
to the axis 
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1848 


select ar 
feature 
instument 
axis mc 


existing 
of the 
to be the 
3rker(s) 



1850 



design the instrument 
to include the axis 
marker{s) 



1852 



align the axis marker(s) 
to the graphic guide(s) 
to align the instru ment 
to the path 



Q end ^ 
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target depth not yet reached taget depth reached 
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aligned with path, aligned with path, 

but not at correct depth and at correct depth 
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Q start ) 



define a point on an actual target and an actual path to reach the point on the 

actual target 



2302 



I. 



track a pose of an instrument with respect to a pose of the actual target 2304 



choose an orientation of the graphical representation around the virtual line of 
sight according to a pose of a user with respect to the actual target 



2306 



determine (based on a selection) the orientation such that East, West, 

North, and South correspond to right, left, forward, and backward, 
respectively, for the pose of the user where user faces the actual target 



dynamically adjust the orientation according to a change of the pose of 

the user 



2306a 



2306b 



dynamically adjust the selection with respect to the pose of the user 2306c 



T 



render a graphical representation of the actual instrument and theactual targe to 
obain a virtual instrument and a virtual target point, respectively, the graphical 
representation being rendered with respect to a vitual viewpoint from which a 
virtual line of sight to the virtual target point coincides with a virtual path or th 
irtual instrument to follow during a positioning of the actual instrument to the 
point on the actual target, the virtual path corresponding to the real path 



2308 



render graphical mformation about a distance betwen the actual instrument and 

the point on the actual target (optionally, according to a virtual camera with a 
wide angle lens) 



2310 



overlay the graphical information about the distance onto the graphical 
representation 



2312 



2314 



2316 



2318 



2320 



combine the graphical representation from the virtual viewpoint with another 
graphical representation from another virtual viewpoint looking at the virtual path 

from the side 



combine the graphical representation from the virtual viewpoint with an 
augmented reality view 



align the virtual instrument along the vitual hne of sight o the virtual target point to 
align he actual instrument along the actual path 



move the actual instrument along the actual path towards the point on the actual 
target 
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FIG. 26 



determine a graphics proximity marker(s) for indicating 
a proximity of a predetermined portion of an instrument 

to a target, proximity based on, e.g., final forward 
instrument position, measures for outer/inner surfaces/ 
portions of target, measures for target front and back 



determine a graphics path marker(s) for identifying at 
least one path for the instrument to the target 



render the graphics proximity marker(s) such that the 
proximity of he predetermined portion of the instrument 
to the target is ascertainable based on a position of a 
marker on the instrument with respect to the graphics 
proximity marker{s) 



render the graphics path marker(s) such that the at 
least one path Is identified by the graphics path 
marker(s) 
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end ^ 
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